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To what extent have USAID programs of this type contributed in a measurable 
way to improved trade capacity in the target countries? 

What impact has USAID TCB projects had on the firms, individuals, associations, 
sectors, economies and government agencies targeted by the interventions? 

f Which activities have been more successful in achieving their objectives, and 
what were the primary factors for their relative success? 

What combinations of activities or interventions were more successful and 
sustainable than others, and what were the primary synergies that contributed 
to that success? 

To what extent have the interventions funded by USAID since 2002 succeeded 
v In accomplishing the program's objectives? 

JJj How can USAID integrate monitoring and evaluation into the design and 
^ implementation of TCB programs more systematically? 
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Research Criteria in RFP 



• Retrospective - from 2002 

• Cluster approach - examine key segments of TCB program 

• Represent and capture geographic variations 

• Representative of range of projects in terms of size and scope 

• Inclusive of both successful and unsuccessful implementation 

• Rely on quantitative approaches to extent possible 

• Explanatory as well as descriptive - factors affecting 
success/impact 



Evaluation Desiqn at Two Levels 
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MACRO 

Clusters - started with topics (in U.S. TCB Database) but 
shifted to Results Framework - a stronger approach in 
hypothesis testing terms 

Scale of research effort - started with all agencies; narrowed 
to USAID, narrowed again to "directly trade related" projects 
- searched for all cases; captured 70% in value terms, 
including most large, multi-year projects (n = 256) 

MICRO 

* Methods on a Question by Question basis - MSI "Getting 
to Answers" table forces question specific approaches 

Challenges: -- Recovering the past 

-- Counterfactual 
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Evaluation Methods 



"Getting to Answers" 



Evaluation 
Questions 


Type of Answer/ 
Evidence Needed 
(description; 
comparison (to 
what); 
cause and effect 
(and notes on any 
special 
requirements 
or sources of 
data) 


Methods for Data Collection, 

e.g., Records, Structured Observation, Key 
Informant Interviews, Mini-Survey 


Sampling or 
Selection 
Approach, 

(if one is 
needed) 


Data Analysis 
Methods, e.g., 
Frequency 
Distributions, Trend 

Analysis, Cross- 
Tabulations Content 
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Analysis 


Method 


Data Source 
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Question 

To what extent have USAID programs of this type contributed in a 
measurable way to improved trade capacity in the target countries? 

Methods 

• Evidence from projects - contemporaneous reporting on results 
at multiple levels of the Results Framework (Phase II); follow-up 
with participant/observers (Phase III) 

• Independent regression analysis of impact of TCB funding on key trade 
performance measure at four levels of the Results Framework. 

• Separate analysis of matrix of recipients of TCB assistance from 
USAID (high, low, none {counterf actual) by their export performance 
(high, low) - identify factors associated with high exports. 

• Traceback in high export countries (regardless of TCB funding level) 
to projects 
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Evaluation Methods 



Analysis of project reports 
and the regression analysis 
focused on the man of the 
same performance measures 



RF2.1 

Firm/Industry/Sector 

Export/Import & 
Investment Attraction 
Practices Improved 



New products/ 
new markets 
Quality of exports 
increased 



RF1.1 

Trade Performance/ 
Foreign Investment 

Improved in 
TCB Target Countries 



Exports Increased 
Imports Increased 
FDI Increased 



RF2.2 

Trade- Related 
Public Sector 
Practices Improved 



RF2.3 

More Efficient/Cost- 
Effective 
Movement of Traded 
Goods Across Borders 



Visible chantes in 
practices, e.g., 
lower tariffs 
Behind the scenes 
changes in procedures 



Time to export/import 
reduced 

Cost to export/import 
reduced 
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Evaluation Methods 



Question 

What impact has USAID TCB projects had on the firms, individuals, 
associations, sectors, economies and government agencies 
targeted by the interventions? 



Methods 



Evidence from projects at 
program Goal level 

Evidence from projects about firms 
and associations at lower levels of RF 



RFO.O 

Rapid, Sustained, Broad- 
Based Economic Growth 
In TCB Target Countries 



Jobs added 
Incomes increased 



• Analysis of existing data series and information gathered for the 

previous question at the sector level and for economies as a whole. 
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Evaluation Methods 



Data on firms were reported 
at several levels for the 
Private Sectors Practices 
segment off the Results 
Framework 



RF2.1 

Firm/Industry/Sector 

Export/Import & 
Investment Attraction 
Practices Improved 



RF 2.1.1 

Firm/Industry 
Knowledge 
of International 
Market Opportunities 
Increased 



RF2.1.3 

Basic Business Practices 
of Firms/Industries 
Improved 



RF2.1.2 

Firm/Industry 
Knowledge 
Of Trade 

Requirements 
Increased 



RF 2.1 .1.1 

Services from Local 
Export Promotion/Investment 
Attraction Business Support 
Organizations Improved 



RF 2.1 .1.2 

Firm/Industry ICT 
Capacity/Use 
Improved 
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Evaluation Methods 



Question 

Which activities have been more successful in achieving their objectives, 
and what were the primary factors for their relative success? 



Methods 



Requires a success measure. Evaluation adopted existing USAID 
system and made it quantitative to facilitate comparisons between 
projects at RF levels, by geographic location, by sector, etc. 



Did Not 
Meet 

(1) 



Improved, but 
Did Not Meet 

(2) 



Met/ 
Exceeded 

(3) 



• Projects scored at every RF level where they indicated intent; overall 
project rating was an average of scores at all RF levels for a project. 
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Evaluation Methods 



231 of the 256 projects could be scored using success scale. 

Once an average was constructed, subsets of the population 
were compared to each other, and to the overall average. 

2 2.25 2.5 2.737 2.75 3.00 

1 1 

Average Project Success Score 
USAID TCB projects performed well on this measure. 



• Content analysis of project narratives used to extract and 
characterize keys and barriers to success 
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Evaluation Methods 



Question 

What combinations of activities or interventions were more successful 
and sustainable than others, and what were the primary synergies that 
contributed to that success? 

Methods 

• Venn diagrams used to identify combinations 

• Success scores used to compare combinations and identify 



synergies 



• Content analysis of project narratives used to extract and 
characterize synergies 
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Combination of approaches scored higher than either approach alone. 



Cluster 
Approach 

(N= 35) 




Value Chain 
Approach 

(N= 50) 



Synergy suggested 
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Question 

To what extent have the interventions funded by USAID since 2002 
succeeded In accomplishing the program's objectives? 

Methods 

• Primary comparison was to USAID 2003 TCB Strategy Paper 

• Secondary comparisons to: 

USAID country strategies 

Broad goals of the Doha Round embedded in the 
Ministerial Declaration. 
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Question 

How can USAID integrate monitoring and evaluation into the design and 
implementation of TCB programs more systematically? 

Methods 

• Phase III focus on current practices. Phase III will focus on 
approaches for improving TCB M&E. 

• Documentation of indicators used at each RF level in real projects 
including frequency distributions at each RF level - de facto "votes" 
for indicators field can actually use. 

• Project evaluations located and rated using two existing MSI tools: 

Evaluation Statement of Work Checklist 
Evaluation Report Checklist 



